Incident Report: Unable to retrieve logs in the OrderPayable events collector

Date: 2024-03-14
Time: 00:22 (UTC+2)
Duration: 11 minutes

Description

There were 2 of Unable to retrieve logs in the OrderPayable events collector on bsc and on rsk alerts which are open for more than 15 minutes

Root Cause

We were falling back to public RPCs on those chains (the reblok requests probably failed temporarily).

Impact

If the collectors are unable to collect OrderPayable events, we won't be able to see if we've received an order for managed feeds on that chain.

Timeline

00:07 - First noticed the issue.
00:22 - Initial diagnosis.
00:22 - Started the fix.
00:33 - Issue resolved.

Lessons Learned

If we know that the chain is temporarily unavailable, there is nothing much we can do on our side.

Actions Taken

If these alerts persist for an extended duration (e.g. over 30 minutes), we should check if there is an issue with the providers. Vekil identified the problem. Then the alerts auto-closed.

Slack escalation link

Incident Reviewer(s)

Abdel, Mertcan, and Vekil

Description​

Root Cause​

Impact​

Timeline​

Lessons Learned​

Actions Taken​

Related Images/Logs​

Incident Reviewer(s)​